rustdoc: add option to calculate "documentation coverage" #58626

QuietMisdreavus · 2019-02-21T22:14:02Z

This PR adds a new flag to rustdoc, --show-coverage. When passed, this flag will make rustdoc count the number of items in a crate with documentation instead of generating docs. This count will be output as a table of each file in the crate, like this (when run on my crate egg-mode):

+-------------------------------------+------------+------------+------------+
| File                                | Documented |      Total | Percentage |
+-------------------------------------+------------+------------+------------+
| src/auth.rs                         |         16 |         16 |     100.0% |
| src/common/mod.rs                   |          1 |          1 |     100.0% |
| src/common/response.rs              |          9 |          9 |     100.0% |
| src/cursor.rs                       |         24 |         24 |     100.0% |
| src/direct/fun.rs                   |          6 |          6 |     100.0% |
| src/direct/mod.rs                   |         41 |         41 |     100.0% |
| src/entities.rs                     |         50 |         50 |     100.0% |
| src/error.rs                        |         27 |         27 |     100.0% |
| src/lib.rs                          |          1 |          1 |     100.0% |
| src/list/fun.rs                     |         19 |         19 |     100.0% |
| src/list/mod.rs                     |         22 |         22 |     100.0% |
| src/media/mod.rs                    |         27 |         27 |     100.0% |
| src/place/fun.rs                    |          8 |          8 |     100.0% |
| src/place/mod.rs                    |         35 |         35 |     100.0% |
| src/search.rs                       |         26 |         26 |     100.0% |
| src/service.rs                      |         74 |         74 |     100.0% |
| src/stream/mod.rs                   |         49 |         49 |     100.0% |
| src/tweet/fun.rs                    |         15 |         15 |     100.0% |
| src/tweet/mod.rs                    |         73 |         73 |     100.0% |
| src/user/fun.rs                     |         24 |         24 |     100.0% |
| src/user/mod.rs                     |         87 |         87 |     100.0% |
+-------------------------------------+------------+------------+------------+
| Total                               |        634 |        634 |     100.0% |
+-------------------------------------+------------+------------+------------+

Trait implementations are not counted because by default they "inherit" the docs from the trait, even though an impl can override those docs. Similarly, inherent impl blocks are not counted at all, because for the majority of cases such docs are not useful. (The usual pattern for inherent impl blocks is to throw all the methods on a type into a single impl block. Any docs you would put on that block would be better served on the type itself.)

In addition, --show-coverage can be combined with --document-private-items to get the coverage counts for everything in the crate, not just public items.

The coverage calculation is implemented as a late pass and two new sets of passes which strip out most of the work that rustdoc otherwise does when generating docs. The is because after the new pass is executed, rustdoc immediately closes instead of going on to generate documentation.

Many examples of coverage calculations have been included as rustdoc-ui tests.

r? @rust-lang/rustdoc

QuietMisdreavus · 2019-02-21T22:14:45Z

cc @nikomatsakis This is the "doc coverage" feature i've been working on. I don't remember who else in T-compiler was interested in seeing this, though.

QuietMisdreavus · 2019-02-21T22:19:27Z

One thing i forgot to mention: One sizable hole in this feature right now is the fact that items that came from macros are fully counted. This means things like #[derive(Serialize)], whose implementation is mostly out of the crate author's control, is considered an undocumented trait impl. This is another major reason why trait impls were counted separately. I was able to get built-in derive macros filtered out, because they all tag their impls with #[automatically_derived]. Anything from a proc-macro doesn't have this luxury, though.

Centril · 2019-02-21T22:27:02Z

This seems nice. However... as a user of rustdoc, I think I would approach coverage by folder and file rather than by Rust construct. Knowing how many traits are covered compared to how many structs are isn't all that interesting to me... Rather, I would like to know whether a certain module lacks documentation and how substantial the documentation is. A tree structure with folders and files (use a heuristic to decide how much info to show) akin to as seen in https://github.com/cgag/loc with --files would I think be helpful.

GuillaumeGomez · 2019-02-22T00:08:35Z

Another I'd add would be a sentence about adding #![deny(missing_docs)] so users can see where it's missing exactly.

bors · 2019-02-27T18:59:40Z

☔ The latest upstream changes (presumably #58777) made this pull request unmergeable. Please resolve the merge conflicts.

QuietMisdreavus · 2019-02-28T22:19:34Z

Updated and force-pushed. (I was hoping that pushing the commits first before rebasing would preserve their history, but apparently not. >_>) The last two commits are the ones that fix review comments. I've updated the PR description with the new table layout, but i have not updated the tests. I probably need to scrap most of them now that i'm not counting items separately.

QuietMisdreavus · 2019-02-28T22:28:31Z

Pushed an update to the tests. Two things of note:

Since we print the filenames in the output, these tests might be flaky since they're made relative to the current directory if possible. However, since we cut off the name if it's longer than 35 bytes, it may not be a problem, since the names of these files after src/ are all longer than 35 bytes. We'll see how they fare in travis.
Primitives and keywords don't get the right span information when they're cleaned? The exotic.rs test reports two items from the file <anon>, which corresponds to the primitive and keyword which are documented in the test. I'm not sure this is a problem? Since these are only ever used by the standard library, it's not likely to be seen outside of this repo.

src/librustdoc/config.rs

GuillaumeGomez · 2019-03-05T19:59:13Z

src/librustdoc/lib.rs

@@ -391,7 +396,14 @@ fn main_args(args: &[String]) -> isize {
    let diag_opts = (options.error_format,
                     options.debugging_options.treat_err_as_bug,
                     options.debugging_options.ui_testing);
+    let show_coverage = options.show_coverage;
    rust_input(options, move |out| {


Is it necessary to call rust_input in show_coverage case?

rust_input is what actually calls the core::run documentation loading code, so i wanted to go through that to make sure that it used the same cleaning/filtering process that regular docs loading did.

Seems a bit overkill but let's go along for now...

src/librustdoc/passes/calculate_doc_coverage.rs

GuillaumeGomez · 2019-03-05T20:06:41Z

Apart from my comments, looks good to me!

GuillaumeGomez · 2019-03-05T20:49:20Z

Thanks! Once CI passed, r=me.

QuietMisdreavus · 2019-03-05T22:38:25Z

@bors r=GuillaumeGomez

bors · 2019-03-05T22:38:26Z

📌 Commit 3df0b89 has been approved by GuillaumeGomez

…llaumeGomez rustdoc: add option to calculate "documentation coverage" This PR adds a new flag to rustdoc, `--show-coverage`. When passed, this flag will make rustdoc count the number of items in a crate with documentation instead of generating docs. This count will be output as a table of each file in the crate, like this (when run on my crate `egg-mode`): ``` +-------------------------------------+------------+------------+------------+ | File | Documented | Total | Percentage | +-------------------------------------+------------+------------+------------+ | src/auth.rs | 16 | 16 | 100.0% | | src/common/mod.rs | 1 | 1 | 100.0% | | src/common/response.rs | 9 | 9 | 100.0% | | src/cursor.rs | 24 | 24 | 100.0% | | src/direct/fun.rs | 6 | 6 | 100.0% | | src/direct/mod.rs | 41 | 41 | 100.0% | | src/entities.rs | 50 | 50 | 100.0% | | src/error.rs | 27 | 27 | 100.0% | | src/lib.rs | 1 | 1 | 100.0% | | src/list/fun.rs | 19 | 19 | 100.0% | | src/list/mod.rs | 22 | 22 | 100.0% | | src/media/mod.rs | 27 | 27 | 100.0% | | src/place/fun.rs | 8 | 8 | 100.0% | | src/place/mod.rs | 35 | 35 | 100.0% | | src/search.rs | 26 | 26 | 100.0% | | src/service.rs | 74 | 74 | 100.0% | | src/stream/mod.rs | 49 | 49 | 100.0% | | src/tweet/fun.rs | 15 | 15 | 100.0% | | src/tweet/mod.rs | 73 | 73 | 100.0% | | src/user/fun.rs | 24 | 24 | 100.0% | | src/user/mod.rs | 87 | 87 | 100.0% | +-------------------------------------+------------+------------+------------+ | Total | 634 | 634 | 100.0% | +-------------------------------------+------------+------------+------------+ ``` Trait implementations are not counted because by default they "inherit" the docs from the trait, even though an impl can override those docs. Similarly, inherent impl blocks are not counted at all, because for the majority of cases such docs are not useful. (The usual pattern for inherent impl blocks is to throw all the methods on a type into a single impl block. Any docs you would put on that block would be better served on the type itself.) In addition, `--show-coverage` can be combined with `--document-private-items` to get the coverage counts for everything in the crate, not just public items. The coverage calculation is implemented as a late pass and two new sets of passes which strip out most of the work that rustdoc otherwise does when generating docs. The is because after the new pass is executed, rustdoc immediately closes instead of going on to generate documentation. Many examples of coverage calculations have been included as `rustdoc-ui` tests. r? @rust-lang/rustdoc

@ghost

Rollup of 13 pull requests Successful merges: - #58518 (Use early unwraps instead of bubbling up errors just to unwrap in the end) - #58626 (rustdoc: add option to calculate "documentation coverage") - #58629 (rust-lldb: fix crash when printing empty string) - #58660 (MaybeUninit: add read_initialized, add examples) - #58670 (fixes #52482) - #58676 (look for python2 symlinks before bootstrap python) - #58679 (Refactor passes and pass execution to be more parallel) - #58750 (Make `Unique::as_ptr`, `NonNull::dangling` and `NonNull::cast` const) - #58762 (Mention `unwind(aborts)` in diagnostics for `#[unwind]`) - #58924 (Add as_slice() to slice::IterMut and vec::Drain) - #58990 (Actually publish miri in the manifest) - #59018 (std: Delete a by-definition spuriously failing test) - #59045 (Expose new_sub_parser_from_file) Failed merges: r? @ghost

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Feb 21, 2019

QuietMisdreavus added 11 commits February 28, 2019 16:01

add option to calculate documentation coverage

009c91a

tabs -> spaces

9e98a25

count fewer items in calculate-doc-coverage

fc94593

refactor: combine item count numbers into a new struct

95500c0

print doc coverage as a table of individual item types

5eb1ab5

add a coverage mode for private items

a3a2559

tweak wording of extern types

3ce19b4

add tests for doc coverage

63bdd29

update docs for doc coverage

80b4919

track items per-file instead of per-type

1b63543

tweak docs for rustdoc's --show-coverage

74cf1ad

QuietMisdreavus force-pushed the doc-coverage branch from 6df07d9 to 74cf1ad Compare February 28, 2019 22:14

update rustdoc coverage tests with new table layout

515dbe7

GuillaumeGomez reviewed Mar 5, 2019

View reviewed changes

src/librustdoc/config.rs Outdated Show resolved Hide resolved

GuillaumeGomez reviewed Mar 5, 2019

View reviewed changes

src/librustdoc/passes/calculate_doc_coverage.rs Outdated Show resolved Hide resolved

QuietMisdreavus added 2 commits March 5, 2019 14:18

remove unused Display impl

e28cf74

only print coverage pass lists if running on nightly

3df0b89

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 5, 2019

Centril mentioned this pull request Mar 9, 2019

Rollup of 14 pull requests #59043

Closed

Centril mentioned this pull request Mar 9, 2019

Rollup of 13 pull requests #59046

Closed

Centril mentioned this pull request Mar 9, 2019

Rollup of 13 pull requests #59050

Merged

bors merged commit 3df0b89 into rust-lang:master Mar 9, 2019

QuietMisdreavus deleted the doc-coverage branch March 11, 2019 15:19

sanxiyn mentioned this pull request Mar 21, 2019

[rustdoc] Tracking issue for --show-coverage #58154

Open

QuietMisdreavus mentioned this pull request Apr 1, 2019

Document the automatically_derived attribute. rust-lang/reference#555

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustdoc: add option to calculate "documentation coverage" #58626

rustdoc: add option to calculate "documentation coverage" #58626

QuietMisdreavus commented Feb 21, 2019 •

edited

Loading

QuietMisdreavus commented Feb 21, 2019

QuietMisdreavus commented Feb 21, 2019

Centril commented Feb 21, 2019

GuillaumeGomez commented Feb 22, 2019

bors commented Feb 27, 2019

QuietMisdreavus commented Feb 28, 2019

QuietMisdreavus commented Feb 28, 2019

GuillaumeGomez Mar 5, 2019

QuietMisdreavus Mar 5, 2019

GuillaumeGomez Mar 5, 2019

GuillaumeGomez commented Mar 5, 2019

GuillaumeGomez commented Mar 5, 2019

QuietMisdreavus commented Mar 5, 2019

bors commented Mar 5, 2019

rustdoc: add option to calculate "documentation coverage" #58626

rustdoc: add option to calculate "documentation coverage" #58626

Conversation

QuietMisdreavus commented Feb 21, 2019 • edited Loading

QuietMisdreavus commented Feb 21, 2019

QuietMisdreavus commented Feb 21, 2019

Centril commented Feb 21, 2019

GuillaumeGomez commented Feb 22, 2019

bors commented Feb 27, 2019

QuietMisdreavus commented Feb 28, 2019

QuietMisdreavus commented Feb 28, 2019

GuillaumeGomez Mar 5, 2019

Choose a reason for hiding this comment

QuietMisdreavus Mar 5, 2019

Choose a reason for hiding this comment

GuillaumeGomez Mar 5, 2019

Choose a reason for hiding this comment

GuillaumeGomez commented Mar 5, 2019

GuillaumeGomez commented Mar 5, 2019

QuietMisdreavus commented Mar 5, 2019

bors commented Mar 5, 2019

QuietMisdreavus commented Feb 21, 2019 •

edited

Loading